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METHODS  FOR  PROVIDING  DIRECT  FEEDBACK  ABOUT  DECISION  PROCESSES 
FOR  COMMAND  AND  CONTROL  CLASSROOM  EXERCISES 


Introduction 

Many  team  situations  permit  the  participants  to  "play  back" 
recent  occurrences  so  they  can  evaluate  what  they  did  right  and 
wrong.  A  classic  example  is  the  post  mortem  that  usually  takes 
place  immediately  following  the  completion  of  a  hand  of  cards. 

In  the  course  of  this  replay,  the  participants  evaluate  the  cards 
played,  options  taken  and  bypassed,  correct  and  incorrect  plays, 
in  an  effort  to  understand  what  transpired.  One  situation  in 
which  this  luxury  is  rarely  available  is  in  the  course  of  team 
decision  making.  These  situations  are  usually  too  complex  and 
fast  moving  to  permit  the  level  of  detailed  analysis  that 
normally  accompanies  things  such  as  the  replaying  of  a  hand  of 
cards.  Granted,  evaluations  of  the  quality  of  the  decisions 
often  occur,  but  seldom  do  they  touch  on  the  processes  of  the 
decisions.  This  is  what  we  were  attempting  to  accomplish  with 
this  project:  provide  a  means  of  quickly  playing  back  the 
planning  processes  in  such  a  manner  that  they  could  be  seen  and 
evaluated  by  the  participants. 

We  wanted  to  determine  whether  Klein  Associates'  methods  of 
knowledge  elicitation  and  data  collection  would  permit 
examination  of  real-time,  military  team  decision  strategies  such 
that  this  immediate  feedback  to  the  participants  could  be 
provided.  Immediate  feedback,  as  it  does  in  card  playing,  should 
provide  the  military  team  an  opportunity  to  discuss  the  quality 
of  their  performance.  In  addition,  the  feedback  our  methods 
provide  gives  an  objective,  unbiased  basis  for  these  discussions 
which  can  be  used  by  an  instructor  or  trainer  as  well  as  the  team 
members  themselves  to  guide  the  discussion. 

The  concept  of  using  our  knowledge  elicitation  methods  to 
generate  training  feedback  emanated  from  earlier  Klein 
Associates'  projects.  Because  of  this,  some  of  our  earlier  work 
and  the  sequence  of  events  leading  to  this  project  are  worth 
reconstructing . 

Our  initial  studies  of  decision  processes  were  carried  out  in 
the  context  of  examining  proficient  performance.  We  developed 
methods  for  extracting  experts'  tacit  knowledge  and  applied  these 
knowledge  elicitation  techniques  in  a  number  of  domains, 
including  cardiopulmonary  resuscitation  (Klein  &  Klein,  1981) , 
computer  programming  (Peio  &  Klein,  1984) ,  data  analysis  (Klein, 
1985a) ,  and  petro-chemical  control  room  operation  (Klein,  1985b) . 
This  work  led  us  into  a  series  of  studies  more  directly  focused 
on  the  decision-making  process  as  it  occurs  in  real-world 
situations.  Our  first  effort  was  an  investigation  of  decision 
making  by  urban  fire  ground  commanders  (FGCs)  at  the  scene  of  a 
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fire  (Klein,  Calderwood,  &  Clinton-Cirocco,  1988) .  Our  choice  of 
a  data-gathering  method  for  this  study  was  guided  by  our  desire 
to  model  as  closely  as  possible  the  natural  decision  making  of 
FGCs,  while  meeting  the  demands  of  scientific  rigor.  The  method 
chosen  which  we  call  the  Critical  Decision  Method  (CDM) ,  was  a 
retrospective  protocol  analysis  based  on  the  FGC's  reconstruction 
of  his  step-by-step  decisions  and  commands  at  an  incident. 
Incidents  were  chosen  on  the  basis  of  their  having  present  a 
command  challenge .  a  criterion  suggested  by  Flanagan's  (1954) 
critical  incident  method.  The  fact  that  the  reported  incidents 
contained  these  non-routine  decisions  fulfilled  two  major 
requirements  of  the  critical  incident  method  —  that  recall  of 
non-routine  events  tends  to  be  superior  to  that  of  more  routine 
cases,  and  that  the  most  difficult  cases  will  tend  to  reveal 
important  aspects  of  expertise  that  would  not  otherwise  be 
apparent  (Flanagan,  1954) . 

These  studies  have  shown  a  common  form  of  recognitional 
decision  making.  We  have  modeled  the  decision  strategies  we 
observed  as  Recognition-Primed  Decisions  (RPDs) .  (The  model  is 
presented  in  detail  in  Klein,  in  publication.)  These  RPDs  can  be 
considered  to  be  analogous  to  Rasmussen's  (1985)  level  of  rule- 
based  performance.  The  experts  we  studied  were  using  their 
experience  to  judge  the  familiarity  of  problems,  and  to  recognize 
the  typical  way  of  reacting  to  those  problems.  The  decision 
makers  would  evaluate  the  action  that  seemed  to  be  called  for; 
the  extent  of  this  evaluation  seemed  to  depend  on  the  amount  of 
time  available  and  the  presence  of  atypical  dynamics  that  needed 
to  be  considered  more  analytically.  In  other  words,  when  faced 
with  a  decision  point,  our  experienced  subjects  were  able  to 
recognize  the  situation,  recognize  the  typical  reaction,  perform 
some  evaluation  of  the  feasibility  of  that  reaction,  and  then 
carry  it  out.  They  did  not  appear  to  be  doing  any  comparisons  of 
different  options,  looking  for  strengths  and  weaknesses. 

While  we  have  been  studying  expertise  since  1978,  we  had  our 
first  opportunity  to  study  team  decision  making  strategies  in  the 
fall  of  1986  by  observing  Class  I,  wildland  firefighters  at  a 
large  working  fire  in  Idaho  (Taynor,  Klein,  &  Thordsen,  1990) . 
Based  on  results  from  this  study,  we  became  interested  in 
extending  these  examinations  to  team  decision  processes  in  a 
military  command  and  control  environment.  This  was  begun  by 
studying  battalion  level  staff  planners  working  with  the  Army 
Training  Battle  Simulation  System  (ARTBASS)  at  Ft.  Hood,  Texas  in 
the  summer  of  1987  (Thordsen,  Galushka,  Klein,  Young,  &  Brezovic, 
1990) .  One  of  the  outcomes  of  this  project  was  that  we  found  the 
planners  tended  to  employ  a  process  where  they  would  evaluate  an 
option  or  idea  by  gradually  examining  deeper  and  deeper  branches 
of  the  idea  for  workability.  Eventually  they  reach  a  point  where 
the  idea  is  either  accepted,  rejected  or  left  hanging  due  to  some 
distraction.  If  it  is  rejected  the  decision  maker  either  moves 
on  to  a  totally  different  option  or  idea  or  goes  back  up  the 
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deepening  chain  to  a  point  (theoretically)  above  the  source  of 
the  flaw  and  then  follows  another  branch,  once  again  testing  its 
merits  all  along  the  way.  This  process  is  not  unlike  the  concept 
of  progressive  deepening  as  laid  out  by  de  Groot  (1978).  In 
addition,  we  found  we  were  able  to  chart  the  deepening  of  these 
decision  processes  used  by  the  group.  An  example  of  one  of  these 
progressive  deepening  charts  is  provided  in  Figure  1.  A 
limitation  was  that  approximately  three  months  of  intensive  data 
analysis  was  required  before  we  could  construct  these  charts. 
However,  because  of  the  wealth  of  information  captured  through 
the  process  of  making  these  charts,  we  were  interested  in 
determining  whether  the  same  process  could  be  accomplished  to 
provide  feedback  to  participants  in  "near  real  time." 
Consequently,  we  used  the  results  of  the  Fort  Hood  exercise 
analysis  to  begin  developing  data  collection  and  reduction 
techniques  to  greatly  accelerate  the  production  of  these 
progressive  deepening  charts.  These  techniques  are  described  in 
the  Methods  section. 

Based  upon  the  outcome  of  a  two-hour  pilot  study  performed  on 
a  graduate-level  seminar  class  at  Wright  State  University,  Ohio, 
we  felt  we  were  ready  to  try  the  technique  at  the  Command  and 
General  Staff  College  (CGSC) .  Through  the  ARI  Field  Unit  at  Ft. 
Leavenworth,  we  made  arrangements  for  a  field  test  of  the 
technique  to  address  the  following  questions: 

1)  Can  the  team  decision  processes  in  a  real-time  command  and 
control  setting  be  tracked? 

2)  If  yes,  can  direct  feedback  be  provided? 

3)  What  is  the  nature  of  the  feedback  that  can  be  provided? 

4)  How  quickly  can  this  feedback  be  provided? 

5)  Is  the  feedback  considered  to  be  helpful  by  the  students 
and  the  instructors? 

6)  What  are  the  limitations  of  the  application  and  what  can 
be  done  to  increase  the  chances  for  a  successful 
application  of  the  technique? 

Performance  feedback  is  typically  given  in  terms  of  the  rated 
quality  of  some  product  the  students  produce.  Rarely  is  an 
attempt  made  to  give  process  feedback.  We  felt  our  method  could 
provide  objective  feedback  on  how  the  students  went  about 
accomplishing  their  task.  Although  no  "school  solution"  exists 
as  a  criterion  measure  of  process  performance,  describing  and 
discussing  the  process  used  should  help  the  students  learn 
valuable  cause-effect  lessons  concerning  the  relationship  between 
the  way  they  do  things  and  the  quality  of  their  products. 
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Figure  1.  Progressive  deepening  chart  example. 


Method 


Subjects 

The  research  involved  students  from  class  A399,  Advanced 
Warfighting,  at  the  Command  and  General  Staff  College  (CGSC) , 

Fort  Leavenworth.  This  was  an  experimental  class  that  was  being 
offered  for  the  first  time.  In  addition  to  instructing  the 
students  in  advanced  warfighting,  it  was  intended  to  experiment 
with  the  use  of  computerization  in  the  classroom,  to  encourage 
doctrinal  discussions,  and  to  work  with  the  concept  of 
synchronization  of  the  battlefield. 

The  class  was  composed  of  62  students.  During  the  course  of 
the  class,  they  took  part  in  two  ’’Quick  Decision  Exercises"  (QDX) 
and  one  Classroom  Planning  Exercise  (CPX) .  The  QDXs  were  short 
3-4  hour  exercises  during  which  the  students  performed  pre-battle 
planning  tasks  (preplanning) .  The  CPX  was  a  five  day  exercise 
during  which  a  major  battle  was  simulated  using  a  game  board 
technique  controlled  by  the  instructors  and  CGSC  staff.  During 
these  exercises  the  students  were  assigned  positions  on  corps, 
division,  and  brigade  staffs  and  were  responsible  for  pre-battle 
planning,  fighting  the  battle  and  replanning  during  the  battle. 

We  observed  both  QDX  pre-battle  planning  exercises  and  the  second 
through  fifth  days'  fighting  and  replanning  of  the  ongoing  battle 
during  the  CPX. 

Procedure 


Our  task  was  to  observe  the  planning  cells  in  such  a  way  that 
general  feedback  about  their  decision  making  could  be  provided  as 
well  as  more  detailed  information  about  the  nature  of  the 
processes  they  employed.  We  were  able  to  collect  the  data  for 
the  general  feedback  with  fairly  straightforward  observation  and 
note  taking.  Generation  of  the  detailed  process  feedback 
required  more  attention  to  capture  the  correct  balance  of 
planning  content  and  process  during  the  observation. 

Planning  content  in  this  situation  refers  to  the  actual  issues 
and  items  being  addressed  by  the  staffs,  such  as  the  use  of  close 
air  support,  the  placement  of  mine  fields,  the  availability  of 
engineering  support,  and  so  forth. 

Process  refers  to  the  category  of  decision  strategy  to  which 
the  content  belongs.  For  example,  the  statement  "we  could  place 
a  mine  field  at  grid  X"  is  likely  to  involve  the  process  of 
generating  an  option,  while  if  the  commander  states  "place  a  mine 
field  at  grid  X"  it  is  more  likely  that  the  process  involves 
making  a  decision. 
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Tracking  both  content  and  process  was  necessary  so  the 
feedback  could  provide  descriptions  of  the  processes  (e.g., 
making  a  decision)  that  were  directly  anchored  to  specific 
content  examples  ("place  a  mine  field  at  X") . 

Normally  the  best  way  to  capture  the  detail  required  is  to  use 
audio  recordings  of  the  exercises.  Unfortunately  it  takes  too 
long  to  transcribe  the  recordings  to  provide  fast  feedback  and 
the  planning  sessions  were  often  too  noisy  to  allow  clean 
recording.  Therefore,  we  had  to  capture  all  observations  by  hand 
notation. 

A  major  difficulty  with  hand  notation  as  opposed  to 
transcripts  is  that  there  is  no  means  to  reconstruct  anything 
that  may  have  been  missed.  In  addition,  there  are  limits  as  to 
how  much  detail  can  be  captured  effectively  at  any  given  time. 

If  the  content  is  tracked  too  closely,  it  is  easy  to  miss  the 
underlying  process,  and  if  the  process  is  concentrated  on  too 
much,  it  is  very  easy  to  1 jse  track  of  the  content.  There  is  a 
balance  between  content  and  process  that  must  be  achieved  to 
optimize  the  data  collection. 

To  help  achieve  this  balance,  a  notational  format  was  devised 
(Appendix  A)  for  the  detailed  observations.  These  coding  sheets 
had  a  reasonable,  but  limited  space  for  content  information,  ten 
"check"  columns  to  track  process  information,  and  one  column  for 
identifying  who  was  being  observed  at  any  given  point.  The 
limited  space  for  content  forced  the  recorder  to  be  precise  in 
the  content  tracking  while  the  process  checks  forced  the  observer 
to  maintain  concurrent  awareness  of  the  processes  that  were 
taking  place  during  that  particular  phase  of  content. 

The  check  columns  for  the  process  information  were  designed  to 
track  the  following  ten  categories: 

Option  Generation:  The  generation  of  options  and  alternatives 
by  the  planners.  For  example:  "We  might  want  to  blow  this 
bridge  as  soon  as  the  enemy  is  at  xxx." 

Information:  The  introduction  of  information  into  the 
planning  environment.  For  example,  "This  armor  unit  is  now  at 
grid  X,  coming  down  towards  the  city."  The  planners  may 
immediately  act  on  this  information,  store  it  for  future  use, 
or  forget  it.  Nevertheless,  it  was  still  available  for  their 
use  at  one  time. 

Decision:  The  process  of  clearly  making  a  decision.  The 
requirement  of  "clearly"  is  met  when  the  decision  is  made  from 
either  a  position  of  strength  (the  ranking  officer)  or  through 
consensus  (agreement  among  the  team  decision  group) .  For 
example,  the  commander  states,  "Have  the  helicopters  go  up  and 
serve  as  observers  to  call  in  the  artillery,"  or  the  entire 


6 


planning  group  reaches  an  agreement  that  what  they  must  do  is 
to  "nave  the  helicopters  go  up  and  serve  as  observers..."  If 
neither  rank  nor  consensus  are  present  in  an  apparent  team 
decision,  more  often  than  not  the  discussion  reopens  later, 
and  the  apparent  "decision"  wilJ  turn  out  to  have  been  just 
another  option  that  was  under  consideration. 

Appraisal :  ..ny  general  discussion,  debate,  argument,  etc. 

that  serves  to  further  the  state  of  the  plan,  but  does  not 
introduce  additional  information,  make  decisions,  or  generate 
options.  Sometimes  this  includes  "kickin'-  a  dead  horse" 
discussions,  since  these  are  judged  to  help  the  individual 
planners  reinforce  their  mental  mind  sets  of  the  plan.  For 
example,  "O.K.,  say  they  go  North... does  that  appreciably 
change  the  battle?" 

Action;  This,  refers  to  activity  or  a  request  for  activity. 
This  is  usually  in  conjunction  with  information  introduction 
and  option  generation.  For  example,  if  the  operations  officer 
tells  the  intelligence  officer  to  check  with  corps  to  find  out 
where  the  lead  element  of  the  enemy  force  is  located  it 
indicates  an  understanding  of  missing  information  (thus  coded 
as  information)  and  an  instruction  to  locate  the  information 
(action) .  A  similar  coding  for  option  and  action  would  exist 
if  someone  had  been  instructed  to  work  out  the  details  of  a 
particular  option. 

Situation  Assessment  Shift;  Any  change  in  the  planners' 
overall  assessment  of  the  current  situation  to  the  extent  that 
it  requires  a  shift  in  goal  states.  For  example:  A  division 
is  planning  a  counter  attack  when  they  receive  new 
intelligence  indicating  the  enemy  has  advanced  faster  than 
they  expected.  The  division  then  has  to  abandon  the  counter 
attack  plan  because  of  a  lack  of  execution  time. 

Situation  Assessment  Elaboration:  Any  change  in  the  planners 
overall  assessment  of  the  current  situation  that  does  not 
require  a  shift  in  goal  states.  For  example:  A  division  is 
planning  a  counter  attack  when  they  receive  new  intelligence 
indicating  the  specific  identity  of  the  advancing  threat  force 
and  more  detailed  information  about  its  makeup.  The  division 
continues  its  counter  attack  planning,  with  a  greater 
understanding  of  the  force  it  will  be  opposing. 

Simulation  Intrusion:  Anytime  the  participants  knowledge  of 
the  existence  of  the  "game"  or  the  artificiality  of  it  causes 
the  content  discussions  to  focus  on  issues  concerning  the 
simulation.  For  example:  "The  controllers  know  where  our 
strength  is,  so  I  bet  they  don't  send  the  enemy  there... just 
to  train  us... for  the  academics." 
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Problems :  Identification  of  a  potential  problem  or 

contradiction.  This  is  normally  used  in  conjunction  with 
appraisal  and  information.  For  example,  (appraisal  and 
problem) :  What  happens  if  the  9th  goes  North  and  the  79th 

goes  South... will  that  mess  us  up? 

Breaks;  Any  change  in  the  focus  of  the  topic  of  the  planning 
discussion.  For  example:  The  operations  officer  and 
intelligence  officer  are  discussing  mine  field  placement. 

The  fire  support  officer  comes  in  and  asks  about  divisional 
fire  support  assets  and  the  Operations  and  Intelligence 
Officers  begin  discussing  this  issue  with  him.  This  is  a 
break,  because  the  focus  of  the  planning  at  that  point  has 
been  changed. 

This  process  categorization  is  derived  primarily  from  our 
indepth  analysis  of  a  battalion  simulation-driven  planning 
exercise.  We  recorded  the  verbal  interactions  of  the  planners 
and  analyzed  the  transcripts  in  detail.  One  of  the  products  of 
this  analysis  was  the  development  of  process  categories  to 
describe  the  types  of  interactions  (Thordsen,  et  al.).  The 
process  categories  were  further  elaborated  as  a  result  of  our 
observation  of  the  two  Quick  Decision  Exercises  (QDX)  earlier  in 
the  A399  course. 

As  mentioned  above,  the  rationale  for  including  these  check 
items  on  the  form  was  to  help  the  observer  maintain  the  balance 
between  content  and  process.  This  assisted  in  tracking  the  level 
of  detail  necessary  to  construct  the  progressive  deepening 
charts,  identify  general  patterns,  and  track  quantifiable  factors 
without  becoming  so  overwhelmed  that  important  information  was 
missed.  Thus,  the  data  collection  form  was  used  to  generate  three 
of  the  four  types  of  feedback  discussed  in  the  Results  section. 
The  other  type,  the  illustrative  incidents,  were  gathered  from 
general  note  taking. 
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Results 


The  results  obtained  are  presented  here  in  four  sections: 

•  Feedback  of  Illustrative  Incidents 

•  Feedback  of  General  Patterns 

•  Progressive  Deepening  Charts 

•  Other  Quantifiable  Factors 

These  results  are  observations  that  either  were  or  could  be 
presented  to  the  participants  as  "snap-shots"  of  their  recently 
completed  activities.  It  requires  some  participation  on  their 
part  to  then  evaluate  the  pros  and  cons  of  their  actions  to  learn 
from  the  post  mortem  of  the  exercise,  or  more  commonly  called  the 
after  action  review  (AAR) .  The  illustrative  and  pattern  feedback 
can  be  back  briefed  with  approximately  30  minutes  of  preparation 
time.  The  deepening  charts  require  about  two  hours  preparation 
each  and  the  other  quantifiable  factors  require  about  one  day's 
preparation. 

It  is  important  to  note  that  our  observations  were  directed  at 
the  quality  of  the  process  of  their  team  decision  making,  and  not 
the  quality  of  the  tactical  decisions. 

Feedback  of  Illustrative  Incidents 


One  method  of  observation,  as  mentioned  earlier,  used  general 
observation  and  notational  means  and  concentrated  on  compiling 
examples  of  illustrative  incidents  that  transpired  during  the 
planning  sessions.  For  the  final  briefing,  these  incidents  were 
reiterated  by  the  observer  while  the  participants  were  encouraged 
with  questions  and  statements  to  critique  the  incidents.  Some 
examples  of  these  observations  are  presented  and  each  is  followed 
by  a  description  of  the  lesson  that  could  be  learned  as 
determined  by  the  observer. 

Example  1 

Use  of  Resources.  One  individual  playing  a  staff  member  of  an 
armored  calvary  regiment  (ACR)  and  who  felt  positive  about 
using  computer  aids,  showed  the  observer  Moveplan:  an  aid  for 
calculating  and  planning  movement.  He  also  demonstrated  a 
related  aid  on  enemy  movement.  However,  at  a  critical 
moment  in  the  ACR  unit  action  where  it  was  important  to 
determine  whether  the  ACR  or  the  enemy  would  win  a  race  to  the 
Hahne  River,  the  engineer  used  his  thumb  and  pinky  on  a  paper 
map  to  estimate  they  were  "a  finger  spread  away,  maybe  45 
minutes  to  an  hour."  While  this  was  taking  place,  no  one  even 
looked  at  the  Moveplan  computer  terminal,  much  less  thought  of 
turning  it  on  or  using  it. 
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Lesson  1 


This  example  was  used  to  demonstrate  that  even  though  there 
were  tools  at  their  disposal,  they  did  not  consider  using  them 
at  a  time  when  they  may  have  been  most  helpful.  They  may  want 
to  consider  making  sure  all  the  decision  support  systems  are 
turned  on  and  ready  to  use  at  the  beginning  of  each  session. 
The  observer  did  not  critique  this  further  but  left  it  up  to 
the  participants  to  decide  whether  it  was  appropriate  to  use 
the  machine  or  rely  on  their  rules  of  thumb. 

Example  2 

Decision  Making.  At  1457  hours  there  was  a  need  for  a  sudden 
shift  in  plans  at  corps  headquarters.  In  response  to  this 
need,  the  corps  commander  made  a  magic  carpet  ride  (i.e.,  took 
a  one-minute  walk  to  the  ACR  that  would  have  taken  1-3  hours 
by  helicopter  in  real  time)  and  asked  if  the  ACR  could  move 
east  and  defend  at  the  Hahne  River  rather  than  the  Fulda,  thus 
allowing  another  division  to  counterattack.  Basically,  he 
asked  the  ACR  to  change  their  defenses  even  though  there  were 
less  than  two  hours  before  the  major  enemy  attack  would  hit 
them.  [In  this  particular  case,  the  corps  commander  was 
trying  to  find  some  mission  for  the  regiment  rather  than  admit 
to  an  earlier  mission  failure  and  assign  them  a  more 
appropriate  missions  such  as  preparing  to  stop  the  next  threat 
echelon  or  cutting  supply  lines]. 

The  ACR  commander  objected  and  argued  that  it  was  too  late  to 
change  their  plans  at  that  point.  His  operations  officer  also 
argued  that  this  would  leave  a  big  gap  between  the  ACR  and  the 
division  for  the  threat  to  slip  through. 

The  instructor  had  been  observing  all  this  and  asked  the 
operations  officer  to  clarify  whether  in  fact  the  corps 
commander  wanted  him  to  move  two  squadrons  from  the  Fulda  to 
the  Hahne  in  less  than  one  hour.  "What  do  you  think?"  was 
the  question  he  then  posed  to  the  operations  officer. 

The  operations  officer  grasped  at  a  solution  —  "We'll  slow 
them  up  with  artillery."  "Maybe  in  EA  Sally,"  the  regiment 
commander  added  hopefully.  However,  the  operations  officer 
and  the  commander  both  remembered  the  difficulty  of  using 
artillery  without  precise  knowledge  of  the  enemy's  location. 

At  1521  hours  the  operations  officer  abruptly  changed  his 
mind,  decided  not  to  defend  at  the  Hahne  River,  and  ran  off 
(magic  carpet)  to  inform  the  corps  commander.  At  this  point, 
almost  25  minutes  had  gone  by  since  he  agreed  to  move  forward. 
These  were  25  minutes  that  could  have  had  a  major  impact  on 
the  time  management  of  the  corps. 
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Following  all  this,  the  instructor  then  discussed  with  the  ACR 
staff  the  folly  of  moving  from  an  80%  good  solution  (defending 
the  Fulda  River)  to  an  attempted  100%  solution  (defending  the 
Hahne)  when  there  was  little  chance  for  the  latter  to  work  and 
a  great  chance  for  total  failure  due  to  the  time  restrictions. 

Lesson  2 

This  was  presented  as  an  example  of  a  decision  process,  albeit 
a  poor  one.  In  addition,  it  was  pointed  out  that  while  the 
instructor  did  a  superb  job  of  refocusing  the  students  to  the 
tactical  realities  of  accomplishing  the  mission  at  hand,  he 
did  not  address  why  the  commander  accepted  the  mission 
in  the  first  place. 

Example  3 

Reinforcement  of  Lessons  Learned.  A  division  was  making  plans 
to  strike  the  9th  Guards  Tank  Army  (GTA)  in  the  morning  when 
it  became  light.  The  instructor  intervened  and  asked — "What 
is  your  goal  now?"  Their  response  was  "Hold  them  until 
morning."  The  instructor  then  followed  with  a  second 
question — "Who  has  better  night  fighting  ability?"  Their 
response  was  "We  do."  They  immediately  changed  their  plans  to 
begin  aggressive  actions  in  the  evening. 

Lesson  3 

While  the  contradiction  in  the  example  was  obvious  at  the  time 
to  the  students,  it  was  repeated  to  them  in  the  briefing  as  a 
means  of  reinforcing  the  lesson  previously  learned. 

Example  4 

Confusion.  Part  of  a  planned  deception  was  to  convince  the 
enemy  that  he  was  experiencing  success  along  certain  parts  of 
the  front.  Early  in  the  battle,  enemy  forces  landed  60 
helicopters  in  the  friendly  rear  area.  There  was  confusion 
about  what  to  do  and  various  options  were  examined,  but  the 
problem  was  that  the  commander  was  not  sure  what  he  wanted 
to  have  happen.  The  source  of  confusion  involved  the  two 
somewhat  contradictory  goals  of  the  deception  plan —  1)  to 
give  the  enemy  success  and  lure  him  in,  and  2)  to  prevent  the 
enemy  from  getting  too  strong.  In  other  words,  there  had  to 
be  a  precise  calibration  of  how  much  success  it  was 
appropriate  to  allow  the  enemy  to  experience.  It  was  a 
delicate  balancing  act. 
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Lesson  4 


This  example  brings  up  strong  tactical  considerations,  but  the 
reason  for  briefing  it  to  the  unit  was  to  provide  them  an 
analogue  that  they  could  refer  to  in  future  similar 
situations.  The  corps  and  the  division  were  unclear  about  how 
much  force  to  apply  and  retrospectively,  this  balancing  act 
was  too  hard  for  these  students  to  pull  off  (the  deception 
failed) .  They  would  have  been  better  off  just  fighting  hard. 
But,  in  spite  of  these  issues,  the  real  problem  was  the 
confusion  caused  by  their  lack  of  clarification  of  ambiguous, 
subjective  terminology  (e.g.,  what  is  success?)  before  the 
definitions  had  to  be  tested. 

Example  5 

Indecisiveness .  The  corps  staff  was  planning  Combat  Aviation 
Brigade  (CAB)  strike  against  enemy  elements.  However  before 
the  plan  was  completed,  a  lower  echelon  officer  came  in  and 
complained  about  insufficient  aggressiveness  on  the  part  of 
the  corps  commander  and  staff.  (The  enemy  was  expected  to 
cross  the  Fulda  River  around  0600  and  the  friendly  main  effort 
was  going  to  wait  and  hit  them  at  0900.  The  CAB  was  part  of 
the  main  effort) .  The  officer  objected — "Why  let  two  full- 
strength  enemy  divisions  come  barreling  into  you?"  When  he 
left  there  was  consternation  on  the  part  of  the  corps  staff. 

The  corps  commander  complained  about  needing  to  change  the 
plan.  Nevertheless,  he  asked  his  plans  and  operations  people 
to  generate  some  courses  of  action  for  review  for  their 
potential  impact,  especially  regarding  the  use  of  artillery 
and  air  assets.  He  considered  taking  out  the  bridges  on  the 
Fulda  River  at  0200  hours.  His  staff  explained  to  him  that 
Air  Force  planes  do  not  fly  at  night;  helicopters  do,  but  they 
work  best  against  moving  targets,  not  stationery  ones.  The 
discussion  noted  that  hitting  a  bridge  would  slow  but  not  kill 
the  enemy  so  they  would  still  have  the  same  number  to  fight. 
The  question  was  raised  about  using  F-lll  fighter  bombers  and 
maybe  using  smart  bombs.  (All  of  this  showing  how  hard  such 
planning  is  when  the  students  lack  some  of  the  basic  knowledge 
about  the  systems  they  are  controlling) .  Later  on,  the  staff 
was  debating  what  to  do... how  to  redirect  the  52nd  Division. 
Eventually  the  plans  and  operations  people  briefed  their 
plans.  (All  this  was  eating  up  critical  time  needed  to 
prepare  for  the  strike  and  to  synchronize  CAB,  electronic 
warfare,  and  various  fire  support  nets) .  Later  yet,  the 
instructor  yelled  at  them  to  get  the  order  out.  Eventually 
they  did  but  it  was  almost  two  hours  after  they  were  diverted. 
In  the  mean  time  the  9th  GTA  had  sped  up  and  by  the  time  the 
helicopters  lifted  off  the  first  of  the  three  target  elements 
had  already  "gone  to  ground",  they  were  no  longer  moving  and 
thus  less  vulnerable.  The  strike  went  on  against  the 
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remaining  two  target  elements  and  only  20  tanks  were  reported 
hit.  The  strike  was  judged  a  failure. 

Lesson  5 

This  example  was  used  to  demonstrate  graphically  to  the 
students  how  easily  they  got  caught  up  in  their  attempt  to 
please  the  dissident  officer  at  the  expense  of  time 
management.  The  corps  commander  indicated  afterwards  in  an 
interview  that  he  went  through  with  the  assessment  of  the 
other  two  courses  of  action  just  to  comply  with  what  the 
dissident  officer  wanted.  Ironically,  the  plan  he  selected 
was  the  original  plan  they  were  working  on  when  the  officer 
came  in.  The  plan  failed,  largely  because  of  the  delay  in 
execution. 

Example  6 

Hidden  Ideas.  A  CAB  staff  member  made  the  tongue-in-cheek 
comment  "We're  just  waiting  to  get  hit  so  we'll  know  where  to 
f ight . " 

Lesson  6 

Even  though  this  was  said  sarcastically,  it  may  have  been  a 
good  idea.  That  is,  it  may  have  been  appropriate  to  send 
units  east  of  the  Hahne  and  Fulda  Rivers  to  make  contact  with 
the  enemy  and  determine  the  approach  they  were  taking. 

Example  7 

Knowledge.  Out  of  curiosity  about  an  approaching  threat 
force,  the  observer  asked  a  staff  officer  about  the  relative 
force  strengths.  The  officer  initially  tried  to  calculate  the 
number  of  troops  in  each  division  and  then  stopped  and  said  it 
was  really  a  question  of  the  number  of  vehicles.  With  the 
help  of  another  staff  member  he  then  estimated  270  vehicles 
for  the  79th  enemy  division  and  started  to  tally  his  own 
resources  but  never  completed  the  task. 

Lesson  7 

This  example  pointed  out  that  the  lack  of  relative  force  ratio 
knowledge  and  implied  that  they  could  not  have  been 
systematically  working  out  a  synchronized  defense.  The  force 
ratio  knowledge  would  have  been  necessary  to  determine  the 
the  proper  defensive  posture  and  placement. 
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Example  8 


Communication  and  Rumors.  A  unit  was  questioned  via  radio 
about  whether  enemy  forces  were  attacking  the  eastern  edge  of 
their  headquarters  area.  As  this  question  was  repeated  among 
the  staff  it  was  reinterpreted  as  a  factual  piece  of 
information.  At  this  point,  the  staff  became  extremely 
disgusted  that  the  enemy  had  slipped  past  the  defenses  and  was 
upon  them.  "Forget  all  our  plans,"  someone  said.  "This  was 
another  case  of  the  untrustworthy  computer  system."  However 
they  soon  realized  that  the  enemy  was  not  upon  them  and 
nothing  implausible  had  happened.  They  had  twisted  a  simple 
question  into  a  rumor  they  themselves  had  created. 

Lesson  8 

In  this  particular  case  the  inaccurate  information  was 
corrected  before  any  real  damage  was  done,  but  it  is  a  good 
example  of  how  easily  information  can  be  twisted  and 
misrepresented.  This  type  of  incorrect  information  could  be 
the  source  of  major  problems  in  other  situations. 

Example  9 

Synchronization  and  Timing.  It  was  1415  hours,  and  an  armored 
calvary  regiment  was  maneuvering  to  get  to  the  Fulda  to  defend 
against  the  enemy.  They  anticipated  the  enemy  would  arrive 
there  in  one  or  two  hours.  At  1421  hours  the  operations 
officer  started  asking  for  the  location  of  the  lead  enemy 
echelon  so  an  aviation  attack  could  be  launched  at  them.  He 
wanted  to  use  suppression  of  enemy  air  defense  (SEAD)  and 
attack  helicopters. 

The  intelligence  officer  was  not  able  to  pinpoint  the  lead 
enemy  elements.  The  operations  officer  then  told  the 
intelligence  officer  to  get  this  information  from  corps.  In 
the  meantime,  the  operations  officer  began  drawing  engagement 
areas  (EA)  and  considered  moving  up  maneuver  companies.  All 
the  while  he  was  fighting  against  the  lack  of  time. 

Eventually  he  found  that  one  of  the  EA  he  was  plotting  was  no 
longer  good  because  some  enemy  elements  were  already  there  in 
a  defensive  posture  with  their  anti-air  in  place. 

Lesson  9 

This  example  suggested  that  the  synchronization  training  did 
not  help  them  fully  anticipate  their  needs.  They  worked  the 
maneuver  problems  and  ignored  the  need  to  slow  the  enemy  until 
it  was  too  late. 
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Direct  Feedback  of  General  Patterns 


The  second  observation  method  involved  collecting  information 
that  permitted  feedback  regarding  general  patterns  that  were 
observed.  The  following  items  were  back  briefed  to  a  division 
staff  following  the  CPX  exercise.  Some  of  these  cases  will  be 
similar  to  the  above  illustrative  examples,  and  they  are  repeated 
here  to  show  the  different  approaches  and  perspectives  that  can 
be  used  for  the  briefings.  In  some  cases,  it  may  be  appropriate 
to  use  both. 

Overall:  Generally,  the  division  staff  worked  well  together. 
Teamwork  was  very  effective. 

This  observation  was  made  based  on  the  fact  that  we  had 
tracked  the  various  sources  of  input  and  the  amount  of 
information  that  was  contributed  by  each.  Based  on  these 
observations,  we  noted  that  a  wide  range  of  individuals 
participated  in  the  planning  session  and  the  individuals  who 
were  supposedly  "in  charge"  contributed  the  most.  At  the  same 
time,  input  of  the  other  individuals  was  always  encouraged  and 
given  serious  consideration.  We  have  seen  this  not  work  well 
on  other  occasions  at  a  corps  level  exercise  at  Ft. 
Leavenworth,  a  battalion  ARTBASS  exercise  at  Ft.  Hood,  Texas, 
and  at  the  company  and  platoon  level  at  Ft.  Knox,  Kentucky. 

Communications :  The  communications  and  sources  of  input  into 
the  plan  were  very  diverse  and  appeared  to  be  in  desirable 
proportions.  For  example,  over  the  course  of  5  hours  and  7 
minutes  of  observation  on  one  day,  input  from  seven  major  and 
eleven  minor  participants  was  observed. 

There  also  appeared  to  be  a  fairly  good  balance  of  input  from 
these  multiple  sources.  The  proportions  from  each  appeared  to 
be  in  line  with  the  responsibilities  inherent  in  each  of  the 
positions  for  the  task  at  hand.  This  was  determined  by 
deriving  estimates  through  a  quick  analysis  of  the  data  on  the 
observation  forms.  A  more  detailed  evaluation  of  these 
figures  is  presented  in  the  results  section  addressing  other 
quantifiable  factors  (e.g.  Tables  2  and  4). 

Decisions :  We  have  come  to  expect  in  team  decision  settings 

that  there  will  be  few  clear  identifiable  decisions.  We  first 
noticed  this  while  collecting  data  on  wildland  firefighters 
(Taynor,  Klein,  &  Thordsen,  1987)  and  later  during  the 
analysis  of  the  data  from  the  battalion  ARTBASS  exercise  at 
Ft.  Hood,  Texas  (Thordsen,  et  al.). 


15 


While  an  individual  decision  maker  such  as  an  urban  fireground 
commander  tends  to  make  many,  rapid  decisions,  the  process 
appears  to  develop  differently  in  the  team  decision  setting. 
These  team  "decisions"  tend  to  be  options  that  are  deepened  by 
different  individuals  in  the  group  until  they  reach  levels 
that  are  adequate  for  the  task  at  hand.  This  is  what  we 
noticed  in  this  exercise  as  well.  The  result  is  basically  a 
decision;  however,  if  you  asked  the  individuals  involved, 
there  is  a  good  chance  no  one  individual  in  particular  will 
feel  he  or  she  is  the  one  who  made  the  decision. 

In  addition,  most  deliberation  of  options  was  serial,  i.e., 
concurrent  comparison  of  multiple  options  did  not  take  place. 
Generally,  an  option  was  selected  and  improved  upon  until  it 
was  functionally  acceptable  or  if  it  could  not  be  made 
acceptable,  it  would  be  rejected  and  they  would  begin  working 
on  another  option.  Only  one  occasion  of  concurrent  option 
comparison  was  noted  during  one  particular  5  hour  and  7 
minutes  period  of  planning. 

Simulation:  All  in  all,  the  game  board  simulation  seemed  to 

create  very  few  major  problems.  There  were  a  few  occasions 
when  elements  of  the  simulation  (artificiality,  etc.)  crept 
into  the  discussion  and  diverted  the  work  for  short  periods  of 
time,  but  these  were  relatively  rare.  We  were  able  to 
directly  track  these  from  the  coding  sheets  since  we  were 
watching  for  specific  occasions  where  the  simulation 
interfered  with  the  overall  purpose  of  the  exercise. 

The  most  disruptive  factor,  as  appears  to  often  be  the  case, 
is  "time  warping."  This  takes  place  when  the  time  of  the 
battle  is  altered  in  any  manner  that  violates  real  time. 
Fortunately,  time  was  not  warped  by  the  controllers  to  any 
extreme  in  this  exercise.  When  warping  did  occur  (and 
sometimes  when  it  did  not)  it  served  as  an  easy  "scapegoat" 
for  anytime  the  students  were  not  able  to  adequately  track  the 
threat's  progress.  On  the  other  hand,  the  students  themselves 
often  time-warped  via  their  "magic  carpet  rides"  around  the 
battlefield.  This  is  a  situation  where  an  individual  will 
walk  across  the  room  or  hall  to  talk  to  some  other  person  and 
be  back  in  their  headquarters  in  five  minutes  for  "trips"  that 
would  take  2-3  hours  in  real  time  on  the  battlefield.  In  the 
long  run,  these  practices  can  have  implications  regarding 
training.  If  the  actual  time  is  abbreviated,  the  natural  flow 
of  the  process  has  to  be  compressed  or  circumvented  to  "catch 
up"  to  the  accelerated  time  location.  Conversely,  if  "time 
outs"  are  inserted,  the  participants  will  often  do  additional 
planning  and  processing  that  they  would  not  normally  have  the 
time  to  accomplish.  This  can  generate  confusion  for  the 
students.  On  the  other  hand,  the  point  can  be  argued  that 
some  forms  of  training  may  require  that  the  time  flow  be 
altered.  The  optimal  decision  as  to  time  warping  depends  on 
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the  training  goals  of  the  program.  Nevertheless,  it  is  a 
decision  that  should  always  be  addressed  because  of  its 
potential  impact  (pro  and  con)  on  the  training  experience. 

Use  of  Computer  Assistance;  Use  of  automation  seemed  to  be 
primarily  limited  to  the  Maneuver  Control  System  (MCS) .  The 
students  appeared  very  willing  to  use  the  system;  however, 
they  appeared  to  need  more  practice  with  the  system  so  they 
could  be  familiar  with  all  of  its  strengths  and  limitations. 

A  main  issue  of  confusion  concerned  the  difference  between 
updating  the  data  base  versus  sending  message  alert  updates 
via  the  electronic  mail  function.  Even  where  they  got  past 
this  problem,  there  was  still  confusion  about  who  was  to  do 
the  actual  updating  of  the  data  base.  There  was  much  emphasis 
in  the  class  upon  using  the  computer  aids,  and  not 
surprisingly,  when  you  asked  the  students,  most  spoke  very 
highly  of  the  computer  systems  and  yet  they  did  not  use  them 
during  the  exercise. 

Synchronization  Matrix:  The  synchronization  matrix  was 
developed  at  the  Command  and  General  Staff  College  as  a 
means  of  graphically  portraying  the  temporal  relationships 
among  the  actions  required  by  the  various  battlefield 
operating  systems  in  executing  a  plan.  The  results  of  working 
with  the  synchronization  matrices  was  evident.  The  students 
appeared  to  have  a  good  feel  for  the  overall  battlefield.  The 
students  seemed  to  be  very  aware  of  the  fact  that  updating  the 
matrix  during  the  actual  battle  was  not  feasible.  However, 
while  using  the  entire  matrix  is  not  realistic  during  the 
battle,  they  might  find  it  helpful  to  pull  out  key  elements  of 
it  to  use  as  checklists  during  their  replanning,  to  help 
assure  they  are  taking  key  points  into  consideration.  As  good 
as  the  matrices  are,  we  do  urge  caution  in  that  they  have  the 
potential  to  lull  one  into  a  false  sense  of  security  due  to 
the  great  amount  of  detail  they  require  to  be  successfully 
successfully  completed.  This  amount  of  detail  can  misguide 
one  into  believing  you  have  covered  everything,  when  in  fact, 
there  will  always  be  some  things  that  are  missing.  Another 
point  of  caution  is  that  this  same  amount  of  detail  can 
potentially  result  in  an  information  overload  condition  where 
there  is  so  much  detail  that  critical  material  gets  buried  in 
the  volume.  In  other  words,  there  may  be  a  need  for  training 
the  students  to  down-size  the  matrix  to  a  manageable  level 
based  upon  the  situation. 

Information  Seeking  and  Acquisition.  The  students 
communicated  very  well  within  their  staff.  They  did  not  do  as 
well  if  they  had  to  actively  seek  out  information  from  outside 
sources.  This  was  relatively  easy  to  track  by  observing  when 
they  indicated  a  lack  of  information  (e.g.,  "Do  we  know  the 
resolution  of  this  battle  yet?")  and  watching  whether  anyone 
actively  tracked  down  this  missing  information.  In  this 
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particular  exercise,  instead  of  actively  seeking  information, 
the  participants  waited  for  the  reports  on  the  resolution  of 
specific  battles  until  long  after  this  information  was 
required. 

This  behavior  is  not  unusual.  We  have  noted  the  same  thing  in 
other  military  planning  session  we  have  observed  (Thordsen, 
et  al.).  Players  did  not  actively  seek  needed  information  jf 
it  was  not  close  at  hand,  even  though  the  consequences  could 
have  been  devastating.  The  bottom  line  is  that  they  have  to 
be  much  more  aggressive  in  their  acquisition  of  information. 

Time  Management.  A  lot  of  the  planning  was  taking  place  very 
close  to  deadlines.  For  example,  they  planned  a  counter 
attack  where  the  key  attacking  force  had  to  be  moving  within 
30  minutes  to  have  any  chance  of  hitting  the  threat's  flank. 

It  is  very  easy  to  be  overtaken  by  events  when  the  planning 
cell  is  not  looking  far  enough  ahead.  This  can  be  tricky, 
because  the  natural  inclination  is  to  wait  until  you  have  all 
the  information  necessary  to  create  a  relatively  risk  free 
plan.  The  problem  is  that  in  a  battle  situation,  you  will 
seldom  be  able  to  collect  all  the  needed  information  to 
develop  a  risk  free  plan  and  still  have  time  to  implement  it 
as  well.  They  have  to  understand  that  they  will  often  have  to 
plan  with  uncertain  and  missing  information  if  they  hope  to 
have  enough  time  remaining  for  a  successful  execution. 

Knowledge  of  Task  Responsibilities  and  Duties.  An  important 
aspect  of  the  makeup  of  this  particular  "division  staff"  was 
that  they  were  generally  working  in  their  natural  roles.  For 
example,  the  operations  officer  was  in  fact  a  combat  arms 
officer.  We  have  observed  several  classes  at  Ft.  Leavenworth, 
and  generally  they  have  the  students  switch  roles  during  the 
exercises.  This  tended  to  result  in  a  significant  amount  of 
confusion  on  the  part  of  the  individuals  about  the 
responsibilities  and  duties  they  were  expected  to  perform.  We 
believe  that  the  fact  that  the  individuals  were  fairly  well 
matched  to  the  positions  in  this  case  improved  how  well  they 
worked  together  and  communicated.  Since  they  were  already 
familiar  with  their  jobs,  they  could  concentrate  on  fighting 
the  battle  rather  than  having  to  simultaneously  train 
themselves  in  less  familiar  positions. 

Background .  All  in  all,  the  division  staff  did  not  appear  to 
fall  into  a  common  trap  we  have  often  observed  where  members 
of  the  staff  allow  their  perception  of  the  battlefield  and 
resources  to  be  strongly  dictated  by  their  background,  such 
that  it  becomes  detrimental.  This  can  result  in  a  form  of 
"tunnel  vision"  where  they  effectively  used  resources  they 
were  familiar  with  (e.g.,  armor)  while  simultaneously  ignoring 
less  familiar  but  equally  available  resources  (artillery, 
infantry,  engineering  support,  air,  etc.).  This  staff 


18 


appeared  to  do  a  fairly  good  job  of  using  the  resources 
available  to  them  and  not  just  as  an  afterthought.  This  may 
have  been  due  to  the  synchronization  emphasis  in  Class  A399. 


Progressive  Deepening  Charts 

We  were  able  to  provide  feedback  about  illustrative  incidents 
and  general  patterns  immediately  following  completion  of  the 
exercises.  This  was  a  primary  goal  and  challenge  of  the  study. 
However,  we  also  wanted  to  determine  whether  we  could  generate 
charts  mapping  the  progressive  deepening  patterns  of  the  decision 
makers.  In  this  particular  case  the  questions  were:  1)  Can  we 
generate  progressive  deepening  charts  in  real  time  as  an  exercise 
is  taking  place?,  and  if  not,  2)  Can  the  charts  be  generated 
immediately  following  the  exercise? 

Several  attempts  were  made  to  map  the  deepening  processes  of 
the  decision  makers  as  the  exercises  were  taking  place.  In 
general,  these  real-time  attempts  were  not  very  successful. 
However,  we  did  find  that  we  could  map  the  deepening  process  from 
the  material  on  the  data  collection  sheets  relatively  soon  after 
the  exercise.  We  found  that  one  analyst  charting  20  minute 
segments  of  the  exercise  will  take  approximately  three  hours  for 
the  first  one  and  about  two  hours  for  every  additional  one. 

Two  examples  of  progressive  deepening  charting  are  included 
here  as  Figures  2  and  3.  These  are  both  from  one  division's 
planning  processes  during  the  CPX. 

Figure  2  maps  a  19-minute  section  of  a  general  discussion 
concerning  possible  enemy  actions  and  friendly  alternatives.  It 
is  important  to  note  that  this  discussion  took  place  early  on  and 
was  largely  a  "brain  storming"  session,  focusing  on  identifying 
the  problems  rather  than  attempting  at  that  time  to  fully  develop 
solutions  for  the  particular  problems.  As  can  be  seen,  there  are 
isolated  segments  of  vertical  deepening  but  in  general  the  flow 
is  wide.  Still  there  is  no  concurrent  comparison  of  options 
during  the  entire  time.  As  mentioned,  the  planners  were  not  yet 
attempting  to  develop  solutions  for  any  particular  problems. 
Because  this  is  the  nature  of  the  task  at  hand,  the  charts  would 
be  expected  to  have  more  of  a  horizontal  (wide)  rather  than  a 
vertical  (deep)  flow.  The  same  can  be  said  of  Figure  3,  where 
once  again  they  are  attempting  to  identify  the  problems  rather 
than  resolve  them.  In  this  five-minute  example,  the  deepening  is 
fairly  shallow  and  it  goes  back  and  forth. 

Regardless  of  whether  the  deepening  charts  appear  wide 
(identification  of  problems)  or  deep  (attempts  to  resolve 
particular  problems)  they  are  still  useful  to  help  the 
participants  "replay  the  hand."  In  other  words,  they  can  look  at 
the  charts  and  see  the  options  that  were  generated,  where  and 
when  they  stopped  pursuing  them,  identify  information  that  was 
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presented  during  the  discussions,  follow  what  they  did  with  the 
information  after  it  was  presented  and  when  and  why  the^  changed 
direction  in  their  discussions,  and  so  forth. 


While  these  results  were  not  presented  to  the  participants 
during  the  AAR,  we  feel  they  can  he  generated  quickly  enough  to 
do  so  and  that  the  planners  would  find  them  informative  and 
useful.  Unfortunately,  at  this  time  we  do  not  know  if  there  is 
any  special  expertise  required  for  the  observer-analyst  to  be 
able  to  collect  the  correct  information  and  quickly  create  the 
deepening  charts.  The  two  individuals  who  were  the  primary  data 
collectors  in  this  exercise  have  done  much  knowledge  elicitation 
in  various  content  domains  and  are  quite  knowledgeable  in  the 
principles  and  techniques  of  observation.  They  possess,  however, 
only  limited  domain  knowledge  in  Army  command  and  control  based 
upon  a  few  past  observations  and  limited  reading  on  the  subject. 
Ideally,  we  hope  the  method  can  be  easily  taught  to  interested 
domain  experts. 
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Figure  2.  Progressive  deepening  for  a  19  minute  segment  of  planning. 
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Figure  3.  Progressive  deepening  for  a  five  minute  segment  of  planning 
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Quantifiable  Factors 


In  addition  to  the  direct  feedback  and  progressive  deepening 
charts,  we  are  able  to  provide  some  quantitative  data  concerning 
other  aspects  of  the  planning  process.  This  information  captures 
more  discrete  elements  of  the  activities  rather  than  following 
the  flow  of  the  planning  as  we  have  seen  in  the  previous  results. 
It  serves  to  provide  a  different  type  of  feedback  for  the 
participants. 

Table  1  shows  the  frequency  of  occurrence  by  type  of  process 
for  the  same  19  minute  period  that  is  mapped  in  the  Figure  2 
progressive  deepening  chart.  As  can  been  seen  from  this  table, 
over  97%  of  their  effort  involved  discussion  (appraisals) , 
treatment  of  information,  and  the  generation  of  options.  There 
were  no  decisions,  situation  assessment  (SA)  shifts  or 
elaborations  during  this  period.  While  this  may  at  first  seem 
surprising,  keep  in  mind  that  this  segment  occurred  early  in  the 
planning  while  they  were  still  trying  to  identify  the  problems 
rather  solve  them.  With  this  understanding,  the  values  do  not 
seem  unusual . 

Table  1 

Process  Categories  for  the  19  Minutes  of  Progressive  Deepening 
from  Figure  1 


Process  Categories  Number 

Occurring 

Percent 

Appraisal 

20 

51.2% 

Information 

14 

35.9 

Option  Generation 

4 

10.3 

Action 

1 

2.6 

Decisions 

0 

0.0 

Situation  Assessment  Shifts 

0 

0.0 

Situation  Assessment  Elaborations 

0 

0.0 

Simulation  Intrusions 

0 

0.0 

TOTAL 

39 

100.0% 

Table  2  lists  the  frequency  of  participation  for  the  various 
staff  members  taking  part  in  this  same  19  minute  segment  of 
planning.  Involvement  is  defined  here  as  the  number  of  times  the 
individual  participated  in  the  session  during  the  19  minutes.  As 
can  be  seen  from  the  table,  the  G3  had  the  most  involvement  with 
the  G3-0perations,  the  instructor,  and  the  division  commander 
taking  fairly  active  roles.  Finally,  the  assistant  division 
engineer  (ADE)  had  a  small  involvement  in  the  session.  While  the 
quality  of  each  individual's  involvement  cannot  be  determined 
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from  this  type  of  information,  it  can  be  very  helpful  in 
determining  whether  or  not  the  appropriate  individuals  are 
involved  to  an  extent  commensurate  with  their  positions  and  the 
situation. 

Table  2 

Individual  Participation  for  the  19  Minutes  of  Progressive 
Deepening  from  Figure  1 


Position 


Times  Participated _ Percent  of  Total 


Division  G3 

12 

32.4% 

Division  G3-0perations 

8 

21.6 

Instructor 

8 

21.6 

Division  Commander 

7 

18.9 

Division  ADE 

2 

5.4 

TOTAL 

37 

99.9% 

Table  3  gives  the  breakdown  across  types  of  process  categories 
for  six  hours  and  seventeen  minutes  of  planning  for  one  division 
on  one  day.  Even  over  this  longer  period  of  observation, 
appraisal,  information  introduction,  and  option  generation  still 
comprise  almost  86%  of  the  processes.  There  were  only  ten 
identified  decision  points  during  the  session.  This  is  not 
surprising  to  us  since  in  previous  research  on  team  decision 
making  processes  we  realized  that  the  decisions  are  generally  so 
distributed  that  it  is  rare  that  they  are  clearly  identifiable. 
This  tends  to  hold  true  not  only  for  the  observers  but  for  the 
participants  as  well.  They  will  know  a  decision  has  occurred, 
but  seldom  does  any  one  individual  in  particular  feel  he  made  the 
decision. 
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Table  3 


Process  Categories  for  6  Hours  and  17  Minutes  of  Planning 


Process  Cateaories 

Number  Occurrina 

Percent 

Appraisals 

133 

38.9% 

Information 

127 

37.1 

Option  Generation 

34 

9.9 

Action 

17 

5.0 

Problem  Identification 

12 

3.5 

Decision  Points 

10 

2.9 

Simulation  Intrusions 

6 

1.8 

Situation  Assessment  Shifts 

3 

0.9 

Situation  Assessment  Elaborations* 

- 

- 

Breaks  in  Focus  of  Planning** 

[60] 

- 

TOTAL 

342 

100.0% 

*SA  Elaborations  were  not  coded  for  the  entire  session. 

**Breaks  are  not  included  in  the  N  or  percent  calculations 

Another  interesting  aspect  of  the  planning  process  is  pointed 
out  by  the  Breaks  category  listed  in  Table  3.  A  break  refers  to 
any  change  in  the  current  focus  of  the  planning  session  to  a 
totally  different  focus.  They  can  be  thought  of  as  points  where 
the  subject  is  changed.  Occasionally  these  breaks  are 
intentional  (e.g.,  "OK,  lets  leave  the  mine  field  placement  for 
later  until  we  are  more  certain  about  their  avenues  of 
approach") .  Unfortunately,  this  smooth  transition  is  generally 
the  exception  and  not  the  rule.  That  is,  the  focus  of  the 
planning  is  usually  changed  unintentionally  because  someone  asks 
a  question  or  makes  a  comment  that  is  out  of  the  current  planning 
context,  prompting  everyone  to  immediately  begin  to  address  the 
new  issue  and  leave  the  previous  topic  behind.  Sometimes  the 
planners  never  return  to  the  previous  topic.  During  the  six 
hours  plus  of  planning  that  are  addressed  by  Table  3,  60  of  these 
breaks  were  identified.  The  quick  analysis,  however,  did  not 
allow  us  to  determine  how  many  of  the  topics  were  resumed  later. 
While  breaks  appear  to  be  a  natural  element  in  team  decision 
making  sessions,  we  believe  there  are  benefits  to  be  gained  by 
making  the  participants  aware  of  the  existence  of  this 
phenomenon . 

Table  4  provides  the  breakdown  of  all  individuals  taking  part 
in  the  six  hour  and  seventeen  minute  session.  Participation  is 
defined  here  as  any  time  an  individual  contributes  in  an  overt 
fashion  (i.e.,  vocally,  indicating  by  pointing  to  the  map,  etc. 

As  with  the  results  outlined  in  Table  2,  this  does  not  address 
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the  quality  of  the  involvement  of  the  individuals.  However,  it 
does  help  identify  who  was  involved  and  to  what  extent. 


Table  4 

Individual  Participation  for  6  Hours  and  17  Minutes  of  Planning 


Position 

Times 

Particioated 

Percent  of  Total 

Division 

G3 

90 

27.9% 

Division 

Commander 

62 

19.2 

Division 

G2 

57 

17.6 

Division 

G3 -Operations 

40 

12.4 

Division 

G3-Plans 

15 

4.6 

Aviation 

Bde  Commander 

15 

4.6 

Division 

ADE 

13 

4.0 

Division 

Instructor 

8 

2.5 

Brigade  Instructor 

5 

1.5 

Corps  G3- 

-Operations 

5 

1.5 

Corps  FA 

Bde  Commander 

5 

1.5 

Corps  Aviation  Officer 

2 

0.6 

Unidentified 

2 

0.6 

Corps  G3- 

-Plans 

1 

0.3 

ACR  Commander 

1 

0.3 

Division 

Asst  G3 

1 

0.3 

Division 

G4 

1 

0.3 

TOTAL 

323 

99.7% 

As  noted  above,  these  results  differ  qualitatively  from  the 
three  previously  outlined  types  of  data.  Subjectively,  it  does 
appear  that  benefits  derived  from  these  quantitative  results 
would  require  more  processing  on  the  part  of  the  participants 
since  the  data  are  presented  outside  of  the  context  of  the 
sessions.  Nevertheless,  we  believe  that  this  material  would  be 
very  helpful  to  the  participants.  For  example,  in  the  session 
covered  by  Table  4,  it  can  be  seen  that  the  Division  G3, 
Commander,  G2  and  G3 -Operations,  participated  the  most.  With 
this  information,  you  can  then  ask  yourself  if  these  are  indeed 
the  people  you  hope  would  be  the  primary  participants.  On  the 
other  side  of  the  coin  are  some  of  the  people  who  participated 
fewer  times,  individuals  you  would  like  to  see  take  more  active 
roles?  The  primary  difficulty  is  that  it  requires  a  minimum  of 
one  day  to  generate  this  data  in  addition  to  any  time  invested  in 
generating  the  feedback  for  the  illustrative  and  pattern  material 
and  constructing  the  progressive  deepening  charts. 
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Discussion  and  Conclusions 

In  the  introduction,  six  questions  were  outlined  that  needed 
to  be  addressed.  They  were: 

•  Can  the  team  decision  processes  be  tracked  in  a  real-time 
command  and  control  setting? 

•  If  yes,  can  direct  feedback  be  provided? 

•  What  is  the  nature  of  the  feedback  that  can  be  provided? 

•  How  quickly  can  this  feedback  be  provided? 

•  Is  the  feedback  considered  to  be  helpful  by  the  students 
and  the  instructors? 

•  What  are  the  limitations  of  the  application  and  what  can  be 
done  to  increase  the  chances  for  a  successful  application 
of  the  technique? 

Each  of  these  questions  will  be  individually  addressed  in  this 
section. 

1.  Can  team  decision  processes  be  tracked  in  real-time  command- 
and-control  settings? 

We  were  able  to  successfully  track  most  of  the  team  processes 
taking  place  in  the  second  (QDX)  and  third  (CPX)  exercises  we 
observed.  In  general,  after  the  initial  acclimation  period, 
tracking  the  processes  proved  to  be  fairly  complex,  but 
manageable.  By  the  completion  of  the  three  exercises,  we  were 
confident  that  we  would  be  able  to  apply  the  same  or  similar 
procedures  to  other  command  and  control  settings. 

2.  If  tracking  can  occur  in  real-time,  can  relatively  immediate, 
direct  feedback  be  generated  for  the  participants? 

While  the  basic  tracking  of  the  exercises  was  manageable,  it 
was  more  difficult  to  accomplish  this  tracking  in  a  way  that 
permitted  us  to  compile  it  quickly  into  a  presentation  format. 

The  observational  methods  that  we  have  developed  through  Critical 
Decision  Method  (CDM)  interviews  and  other  field  observation 
projects  were  quite  adequate  in  this  setting.  However, 
additional  notational  methods  and  tools  had  to  be  developed 
during  the  course  of  the  three  exercises  to  format  the  collected 
data  in  a  fashion  that  permitted  relatively  immediate  feedback. 
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3.  What  is  the  nature  of  the  feedback. 


Four  types  of  feedback  were  generated:  illustrative 
incidents,  general  patterns,  progressive  deepening  charts,  and 
other  quantifiable  information. 

The  illustrative  incidents  involved  using  specific  examples  of 
good  and  bad  performance  drawn  from  the  exercise.  By  presenting 
these  incidents  to  the  participants,  we  provided  them  the 
opportunity  to  see  their  work  from  a  third-party  perspective  and 
to  provide  their  own  critiques  of  the  same.  In  addition,  the 
observers  prompted  this  critiquing  by  asking  questions  and 
providing  their  own  observations. 

The  general  pattern  feedback  was  accomplished  by  identifying 
general  patterns  of  behavior  (teamwork,  communication,  decision 
making,  time  management,  etc.)  that  were  evident  from  the  coding 
sheets.  This  material  was  presented  to  and  critiqued  by  the 
participants  during  debriefings.  As  with  the  illustrative 
incidents,  the  observers  also  prompted  the  critiquing  by  asking 
questions  and  providing  their  own  observations. 

The  progressive  deepening  charting  was  accomplished  by 
reconstructing  the  decision  strategies  of  the  team  from  the 
coding  sheets.  This  process  lays  out  each  piece  of  content  that 
is  part  of  the  process.  This  allows  one  to  visually  follow  the 
flow  of  the  option  generation,  information  treatment,  decision 
making,  etc.  of  specific  segments  of  the  planning  sessions. 

Several  other  quantifiable  factors  were  extracted  from  the 
coding  sheets  that  addressed  the  nature  and  frequencies  of  the 
processes  used  and  the  identity  and  level  of  participation  of 
different  participants.  This  information  is  more  discrete  and 
context  free  than  the  other  three  types  of  material  we  were  able 
to  track. 

4.  How  quickly  can  this  feedback  be  provided? 

The  illustrative  and  pattern  feedback  were  generated  within  30 
to  60  minutes  following  the  completion  of  the  exercise.  Feedback 
generated  via  progressive  deepening  charting  were  generated  with 
2-3  hours  preparation  time  per  chart.  The  more  detailed, 
discrete,  non-context  data  were  generated  with  approximately  one 
day's  preparation  time. 

5.  Is  the  feedback  considered  to  be  helpful  by  the  students  and 
instructors? 

No  systematic  means  of  eliciting  acceptance  responses  were 
employed.  Some  informal  positive  feedback  was  obtained  from  two 
of  the  instructors  but  no  quantifiable  answer  to  this  question 
was  obtained. 
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6.  What  are  the  limitations  of  the  application  and  what  can  be 
done  to  increase  the  chances  for  a  successful  application  of  the 
technique? 

During  the  course  of  the  three  observation  trips,  we 
identified  several  factors  that  were  important  to  the  success  or 
failure  of  the  data  collection.  Many  of  these  factors  can  be 
limitations  if  they  are  not  taken  into  account  before  or  during 
the  application. 

Balance  between  content  and  process.  It  is  very  important  to 
record  the  correct  balance  of  content  versus  process  to  be  able 
to  provide  proper  feedback.  There  is  a  fine  line  between  the 
amount  of  mental  effort  the  observer  directs  toward  tracking  the 
content  of  the  task  versus  the  processes  used  by  the  planners. 
Being  aware  of  this  balance  and  finding  the  appropriate  recording 
instruments  to  help  the  observer  walk  this  fine  line  appear  to  be 
very  critical. 

General  domain  knowledge.  A  certain  amount  of  general  domain 
knowledge  is  required  in  advance  to  permit  intelligent 
observation.  Ideally,  this  methodology  could  be  applied  by  any 
individual  in  any  domain  who  is  trained  to  do  the  observation. 
However,  we  have  found  that  at  least  a  basic  knowledge  of  the 
domain  is  required.  This  includes,  at  a  minimum,  an 
understanding  of  the  terminology  and  goals  of  the  group  being 
observed.  The  observer  does  not  have  to  be  an  expert  in  the 
domain.  However,  a  good  comprehension  of  these  "basics"  is 
necessary.  Unfortunately,  at  this  time  we  do  not  have  any 
estimate  of  how  long  it  would  take  a  domain-knowledgeable 
individual  (e.g.,  an  Army  officer)  to  learn  to  apply  the  method 
to  produce  the  products  covered  in  #4  above. 

Specific  situation  knowledge.  A  large  amount  of  specific 
situation  knowledge  is  required  to  accomplish  the  tracking.  Once 
you  have  the  prerequisite  amount  of  general  domain  knowledge,  how 
much  specific  information  about  the  task  being  performed  is 
required  to  successfully  track  the  process?  We  initially 
attempted  to  track  the  processes  of  the  first  QDX  without  a 
detailed  understanding  of  the  exercise  in  which  the  group  took 
part.  This  did  not  work  well  at  all.  In  fact,  without  being 
able  to  follow  the  specific  exercise  closely,  it  was  almost 
impossible  to  track  the  process.  Therefore,  while  only  a  general 
knowledge  of  the  domain  basics  are  required,  a  thorough 
understanding  of  the  specific  task  situation  is  necessary.  It 
appears  that  without  this  knowledge,  a  large  amount  of  time  is 
spent  attempting  to  comprehend  the  content  of  the  planning 
session,  thereby  throwing  off  the  previously  mentioned  balance 
between  observation  of  content  versus  process. 
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Fatigue.  Because  of  the  enormous  amount  of  detail  that  must 
be  tracked,  fatigue  can  become  a  limiting  factor  for  the 
observers.  Every  sentence  or  statement  needs  to  be  condensed  to 
its  essential  content.  The  nature  of  the  process  used  for  each 
of  these  must  be  determined  either  at  that  time  or  shortly 
thereafter  (within  5-60  seconds) .  While  this  identification  is 
taking  place,  the  next  sentence  or  statement  is  usually 
occurring.  The  observation  requires  a  lot  of  "dual  tasking" 
(listening  to  the  content  and  coding  the  process)  on  the  part  of 
the  observer.  Our  experiences  were  that  a  single  observer  can 
maintain  this  level  of  performance  for  approximately  1  hour  and 
15  minutes  before  a  short  break  (10-15  minutes)  is  required.  We 
believe  it  would  require  at  least  two  observers  working  with 
alternating  breaks  to  track  a  long  session  without  interruptions 
in  the  data  collection. 

Real-time  factors.  Observation  of  the  processes  involved  in 
the  planning  tasks  is  very  dependent  on  the  planning  taking  place 
in  real-time.  If  the  exercise  is  "warped"  ahead,  it  can  (and 
usually  does)  disrupt  the  planning  process  to  the  extent  that 
tracking  it  becomes  somewhat  meaningless.  Also,  if  "time  outs" 
are  inserted  where  the  process  picks  up  without  any  time  elapsed 
after  a  substantial  break,  the  tracking  process  also  suffers.  In 
the  former  case,  the  natural  flow  of  the  process  has  to  be 
compressed  or  circumvented  to  "catch  up"  to  the  accelerated  point 
in  time.  In  the  latter,  the  participants  often  have  time  to  do 
additional  planning  (and  processing)  that  they  would  not  normally 
be  able  to  do.  This  additional  processing  usually  takes  place 
outside  the  planning  group  where  it  cannot  be  observed  and 
tracked.  While  these  factors  were  not  as  evident  with  the  A399 
class's  CPX,  they  were  observed  during  the  two  QDXs  and  during  a 
corps  level  exercise  which  took  place  in  November  of  1986. 
However,  as  mentioned  earlier,  the  need  for  real-time  high- 
fidelity  depends  upon  the  training  goals.  There  may  be  occasions 
when  maintaining  real-time  is  extremely  important  and  other  times 
when  the  training  goals  are  such  that  sacrifices  in  this  aspect 
of  realism  are  appropriate. 

Realism.  If  the  exercise  is  not  realistic,  the  participants 
do  not  take  it  seriously  and  do  not  become  as  involved  in  the 
process.  The  CPX  and  the  two  QDXs  did  not  seem  to  suffer  from 
this  problem. 

Observable  actions.  It  is  absolutely  necessary  that  some  form 
of  overt  action  be  occurring.  If  individuals  are  off  to 
themselves,  silently  working  on  their  own  particular  segments  of 
the  plan,  there  will  be  nothing  overt  to  observe  or  track.  In 
other  words,  there  has  to  be  some  degree  of  interaction  and 
discussion  between  the  participants  for  the  observers  to  observe. 
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This  should  not  be  taken  to  mean  that  overt  action  must  be  forced 
but  rather  that  if  overt  action  is  not  normally  present  then  it 
may  not  be  an  appropriate  situation  in  which  to  apply  these 
techniques. 

Non-intrusiveness .  It  is  probably  safe  to  assume  that  there 
is  no  such  thing  as  total  non-intrusiveness.  This  is  especially 
true  if  observers  are  in  the  same  room  with  the  planners,  let 
alone  if  they  can  be  seen  taking  notes  or  are  asking  occasional 
questions.  However,  we  believe  the  intrusiveness  can  be 
minimized  by  spending  some  initial  time  observing  the  general 
"flow”  of  how  the  participants  work  together  and  attempt  to 
become  part  of  that  flow.  This  is  important,  to  allow  the 
observer  to  know  when  to  ask  questions  and  who  to  ask. 
Nevertheless,  the  process  is  something  like  the  concept  of 
merging  with  the  traffic  on  an  interstate.  It  is  important  to 
understand  that  an  outsider  never  has  the  right-of-way  and  is 
always  responsible  to  yield.  Along  another  but  related  line 
intrusiveness  can  be  affected  by  how  well  the  observer  is 
accepted  by  the  group. 

At  this  time,  we  do  not  have  any  suggestions  for  additional 
changes  in  the  collection  instrument  or  procedures.  This  is  not 
to  say  that  changes  will  not  be  appropriate.  We  envision  these 
taking  place  through  an  iterative  process  where  each  application 
will  point  out  additional  means  of  improving  the  technique  and 
instrument. 
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SUMMARY 


Because  of  the  importance  of  feedback  for  training,  it  is 
critical  that  command  and  control  training  programs  formulate 
procedures  for  obtaining  and  communicating  feedback  about  student 
performance.  Even  for  experiential  training  such  as  classroom 
exercises,  it  is  essential  to  present  informed  feedback.  In  the 
area  of  command  and  control,  this  becomes  even  greater,  because 
students  undergoing  classroom  exercises  without  obvious  cause- 
effect  reinforcement  need  understanding  of  both  the  decisions 
that  were  made  and  the  decision  processes  themselves.  It  is 
important  that  students  learn  what  tactics  should  have  been 
chosen,  but  it  is  also  important  to  use  the  exercise  to  show  them 
effective  and  ineffective  decisionmaking  strategies. 

To  accomplish  the  goal  of  providing  decision  process  feedback, 
trainers  need  techniques  to  use  in  real-time  to  offer  feedback  at 
the  conclusion  of  an  exercise,  or  soon  thereafter.  This  project 
was  designed  to  develop  such  techniques. 

The  project  was  successful  in  identifying  four  types  of 
meaningful  feedback  that  could  be  provided  in  a  timely  fashion. 

First,  general  observation  and  notation  allowed  the 
compilation  of  illustrative  examples  that  transpired  during  the 
planning  sessions. 

Second,  specific  differentiation  between  content  versus 
process  observation  permitted  feedback  regarding  general  patterns 
observed  in  the  planning  processes. 

Third,  we  were  able  to  generate  charts  mapping  the  progressive 
deepening  patterns  of  the  decision  makers. 

Fourth,  we  were  able  to  summarize  additional  quantitative  data 
concerning  other  aspects  of  the  planning  process.  This 
information  captured  the  more  discrete  elements  of  the  activities 
such  as  levels  of  participation. 

Together,  these  four  techniques  offer  instructors  a  capability 
of  using  experiential  classroom  exercises  to  teach  effective  team 
decisionmaking  strategies. 
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It  is  important  to  point  out  that  none  of  these  four 
techniques  has  been  carefully  evaluated  to  determine  the 
effectiveness  of  these  types  of  feedback.  The  goal  was  simply  to 
track  the  team  decisionmaking  in  real-time  and  to  develop 
feedback  procedures.  The  ability  to  collect  the  necessary  data 
and  prepare  it  for  feedback  was  proven.  It  will  require 
additional  research  to  assess  the  full  utility  of  the  four 
techniques.  Nevertheless,  this  project  was  an  important 
beginning  to  the  task  of  enabling  instructors  to  improve  the  way 
that  trainees  engage  in  team  decisionmaking. 
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APPLNDIX  A 


CODING  SHEET 


DATE _  LOCATION _ 

CONTENT/COMMENTS 


CODER. 


I  OPTI  INF[  DPI  APP]  SIM  A 


ELB(  SFTl  PRBj 


Option  Generation 
Information 
Decision  Point 
Appraisal 

Simulation  Intrusion 
Action 

Break  In  Focus 

Situation  Assessment  Elaboration 
Situation  Assessment  Shift 
Problem  Identification 
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